Environment map creation using texture projections with polygonal curved surfaces

ABSTRACT

An environment map creation system creates an environment map from one or more images representing an environment. The environment map creation system includes a texture projection generation unit, which produces a texture projection having polygonal curved surfaces as facets. An environment map rendering unit uses the texture projection to create the environment map from the one or more images. Specifically, the environment map creation system determines an image area in the one or more images corresponding to each polygonal curved surface. The polygonal curved surface is colored based on the corresponding image area. The environment map is formed from the polygonal curved surfaces which become texels in the environment map.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application relates to concurrently filed, co-pending applicationSer. No. 09505337, “POLYGONAL CURVATURE MAPPING TO INCREASE TEXTUREEFFICIENCY”, by Hashimoto, et. al., owned by the assignee of thisapplication and incorporated herein by reference.

This application relates to concurrently filed, co-pending applicationSer. No. 09505442, “ENVIRONMENT DISPLAY USING TEXTURE PROJECTION WITHPOLYGONAL CURVED SURFACES”, by Hashimoto, et. al., owned by the assigneeof this application and incorporated herein by reference.

This application relates to concurrently filed, co-pending applicationSer. No. 09505334, “IMAGE COMPRESSION USING TILE DIVISION”, byHashimoto, et. al., owned by the assignee of this application andincorporated herein by reference.

This application relates to concurrently filed, co-pending applicationSer. No. 09505352, “PARTIAL IMAGE DECOMPRESSION OF A TILED IMAGE”, byHashimoto, et. al., owned by the assignee of this application andincorporated herein by reference.

This application relates to concurrently filed, co-pending applicationSer. No. 09505339, “DISPLAYING IMMERSIVE VIDEOS USING TILEDDECOMPRESSION”, by Hashimoto, et. al., owned by the assignee of thisapplication and incorporated herein by reference.

CROSS-REFERENCE TO COMPUTER PROGRAM LISTING APPENDIX

A computer program listing appendix, incorporated herein by reference,is submitted as part of this disclosure. The computer program listingappendix is stored under the file name: “APPENDIX.TXT” residing on onecompact disk.

A portion of the disclosure of this patent document contains material,which is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but other wise reserves all copyrightrights whatsoever.

FIELD OF THE INVENTION

The present invention relates digital imaging. More specifically, thepresent invention relates to using texture mapping to createenvironmental projections for immersive video applications.

BACKGROUND OF THE INVENTION

Texture mapping is typically used to add realism to graphic images.Generally, texture mapping involves mapping a two dimensional image,typically referred to as the texture map, onto an object. The texturemap contains color information for the object. The texture map isdivided into a plurality of texture elements or texels. Texels typicallyprovide color information for the object. The object is divided into aplurality of facets. Each facet is typically a polygon having one ormore picture elements (“pixels”). The vertex of each facet is assigned apair of texture coordinates which index the texture map to choose atexel (i.e., a color) from the texture map. The color of the facet isderived by interpolating between the colors and the vertices of thefacet. Thus, the image of the texture map is reproduced onto the object.

At one time, the processing requirements of texture mapping limitedtexture mapping to professional graphic systems. However, as theprocessing power of microprocessors has increased, texture mappingsoftware has become useable on consumer level computer systems.Furthermore, special graphics processing hardware capable of texturemapping has also become available for consumer level computer systems.Because texture mapping techniques have become feasible on consumerlevel computer systems, texture mapping techniques have been adapted formany different applications.

One use of texture mapping is environment mapping. Environment mappinguses computer graphics to display the surroundings or environment of atheoretical viewer. Ideally, a user of the environment mapping systemcan view the environment at any angle or elevation. FIG. 1 illustratesthe construct used in conventional environment mapping systems. A viewer105 (represented by an angle with a curve across the angle) is centeredat the origin of a three dimensional space having x, y, and zcoordinates. The environment of viewer 105 (i.e., what the viewer cansee) is ideally represented by a sphere 110, which surrounds viewer 105.Generally, for ease of calculation, sphere 110 is defined with a radiusof 1 and is centered at the origin of the three dimensional space. Morespecifically, the environment of viewer 105 is projected onto the innersurface of sphere 110. Viewer 105 has a view window 130 which definesthe amount of sphere 110 viewer 105 can see at any given moment. Viewwindow 130 is typically displayed on a display unit for the user of theenvironment mapping system.

Conventional environment mapping systems include an environment capturesystem and an environment display system. The environment capture systemcreates an environment map which contains the necessary data to recreatethe environment of viewer 105. The environment display system uses theenvironment map to display view window 130 (FIG. 1) to the user of theenvironment mapping system. Typically, the environment capture systemand the environment display system are located in different places andused at different times. Thus, the environment map must be transportedto the environment display system typically using a computer network, orstored in on a computer readable medium, such as a CD-ROM or DVD.

Computer graphic systems are generally not designed to process anddisplay spherical surfaces. Thus, as illustrated in FIG. 2, texturemapping is used to create a texture projection of the inner surface ofsphere 110 onto polygonal surfaces of a regular solid (i.e., a platonicsolid) having sides that are tangent to sphere 110. Typically, asillustrated in FIG. 2, a texture projection in the shape of a cube 220surrounds sphere 110. Specifically, the environment image on the innersurface of sphere 110 serves as a texture map which is texture mappedonto the inner surfaces of cube 220. A cube is typically used becausemost graphics systems are optimized to use rectangular displays and acube provides six rectangular faces. Other regular solids (i.e.,tetrahedrons, octahedrons, dodecahedrons, and icosahedrons) havenon-rectangular faces. The faces of the cube can be concatenatedtogether to form the environment map. During viewing, the portions ofthe environment map that correspond to view window 130 (FIG. 1 and FIG.2) are displayed for viewer 105. Because, the environment map is.linear, texture coordinates can be interpolated across the face of eachcube based on the vertex coordinates of the faces during display.

An extension to environment mapping is generating and displayingimmersive videos. Immersive video involves creating multiple environmentmaps, ideally at a rate of 30 frames a second, and displayingappropriate sections of the multiple environment maps for viewer 105,also ideally at a rate of 30 frames a second. Immersive videos are usedto provide a dynamic environment rather than a single static environmentas provided by a single environment map. Alternatively, immersive videotechniques allow the location of viewer 105 to be moved. For example, animmersive video can be made to capture a flight in the Grand Canyon. Theuser of an immersive video display system would be able to take theflight and look out at the Grand Canyon at any angle.

Difficulties with immersive video are typically caused by the vastamount of data required to create a high resolution environment map andthe large number of environment maps required for immersive video.Specifically, transmission and storage of the environment maps for highresolution flicker-free display may be beyond the processingcapabilities of most computer systems.

Conventional data compression techniques have been used to compress theenvironment maps and reduce the amount of data transmitted or stored forimmersive video. However, the additional processing time required todecompress a compressed environment map may impair the ability of theenvironment display system to process an adequate number of environmentmaps to provide a flicker-free display. Thus, there is a need for acompression and decompression method for immersive videos that minimizesthe processing time required for decompressing the environment map.

The excessive data problem for immersive video is compounded by theinefficiencies of the conventional texture projections used to formenvironment maps. Specifically, although a cubic texture projection canprovide realistic environment views, the cubic texture projection is notvery efficient, i.e., the average amount of environment information perarea is relatively low. The inefficiency of the cubic projection iscaused by the lack of symmetry between the amount of spherical area onsphere 110 mapped onto cube 220. For example, if each surface of cube220 is subdivided into equal square areas as illustrated in FIG. 3., thesquare areas do not map to equal areas of sphere 110. For concisenessand clarity, only cube face 220_1 of cube 220 is discussed in detailbecause each cube face of cube 220 is typically processed in the samemanner. Specifically, in FIG. 3, cube face 220_1 is divided into N²squares of equal area. More spherical area is mapped onto the squaresnear the center of a cube face than the squares near the edge of a cubeface.

The inefficiency of the cubic texture projection is illustrated in FIG.4. FIG. 4 uses a two dimensional mapping of a circle 410 onto a square420. Specifically, a. quarter of circle 410 is mapped onto each side ofsquare 420. arc segments 411-418 of circle 410 are mapped onto linesegments 421-428 of square 420, respectively. Circle 410 is equivalentto sphere 110, square 420 is equivalent to cube 220, a side of square420 is equivalent to a cube face of cube 220, each line segment ofsquare 420 is equivalent to one of the square areas (FIG. 3) of cube220, and each arc length of circle 410 is equivalent to the area ofsphere 110 mapped on an area of cube 220. Like sphere 110, circle 410has a radius of 1. Therefore, the arc length of an arc segment is equalto the angle of the arc segment in radians. Specifically, arc segment414 has an arc length equal to angle A414 in radians. Angle A414 isequal to the inverse tangent of the length of facet 424 divided by theradius of circle 410. Thus, angle A414 and the arc length of arc segment414 is equal to the inverse tangent of 0.25, which equals approximately0.245. Angle A411 and the arc length of arc segment 411 are equal to theinverse tangent of 1 minus the inverse tangent of 0.75, which equalsapproximately 0.142. Thus, the mapping of circle 410 to square 420results in inefficiencies due to the non-uniformity of the mapping.

Similarly, the mapping of sphere 110 onto cube 220 would result inmapping different amounts of spherical area of sphere 110 onto the equalareas of cube 220. For example, if a cube face is divided into 64squares areas, a corner area would be mapped by only 0.0156 steradians(a measure of surface area) of sphere 120. However, a square area at thecenter of a cube face would be mapped by 0.0589 steradians of sphere110. Thus, for the cubic texture projection, the area near the center ofeach face of cube 220 actually provides lower resolution than the squareareas at the corners of each face. To provide the entire. environment ofviewer 105 in a consistent resolution, a display system using the cubictexture projection must typically conform to the lowest resolution areaof the projection. Thus, the higher resolution areas are not usedoptimally, leading to inefficiencies.

In general, the ideal texture projection for environmental mapping woulduse facets that represent identically sized areas of the sphere, as wellas identically shaped areas of the sphere. Furthermore, an equal sizedareas in each facet should map to equal sized areas of the sphere.Moreover, the facets of the ideal texture projection would collectivelycover the entire environment of viewer 105. However, no practicaltexture projection can satisfy all these criteria. As explained above, alow number of facets results in very low resolution display of theenvironment map. Hence, there is a need for an efficient textureprojection for use with environment mapping and immersive videos.

SUMMARY OF THE INVENTION

Accordingly, the present invention provides efficient texture mappingschemes and compression schemes for environment mapping and immersivevideos. In accordance with one embodiment of the present invention,polygonal curved surfaces are used in place of polygons as the facets ofa texture projection. Specifically, a texture projection generation unitforms a texture projection by dividing the environment into a pluralityof initial polygonal curved surfaces. The initial polygonal curvedsurfaces are subdivided to form a plurality of second-generationpolygonal curved surfaces. The second-generation polygonal curvedsurfaces are further divided to form a plurality of third-generationpolygonal curved surfaces. Division of polygonal curved surfacescontinues until a plurality of last-generation polygonal curved surfacesis created. Each last-generation polygonal curved surface becomes afacet of the texture projection. Various division methods can be used todivide a polygonal curved surface. In accordance with one embodiment ofthe present invention, each polygonal curve of a specific generation hasan equal area.

An environment map creation system uses the texture projection formed bypolygonal curved surfaces to create an environment map. The environmentmap creation system includes an environment capture/generation unit thatprovides one or more images that captures the environment of a user. Acorresponding image area on the one or more images is determined foreach facet of the texture projection. Each facet is colored based on thecorresponding image area. Each initial polygonal curved surface of thetexture projection is converted into a two-dimensional polygonal image.The last-generation polygonal curved surfaces becomes pixels or texelsof the two-dimensional image. The two-dimensional images areconcatenated together to form the environment map.

A compression unit is used to compress the environment map and create acompressed environment map. Specifically, a compression unit inaccordance with one embodiment of the present invention divides theenvironment map into a plurality of tiles. Each tile is compressed by atile compressor independently of the other tiles to form a compressedtile. The sizes of the compressed tiles is used to create a header forthe compressed environment map. In one embodiment of the presentinvention, the header contains an offset value for each compressed tile.The offset value provides the starting location of a compressed tilewithin the compressed environment map.

A decompression unit is then used to decompress a subset of relevanttiles of the environment map. The subset of relevant tiles includes alltiles which contain data needed to texture map a view window. The subsetof relevant tiles may also include some tiles which do not have dataneeded to texture map the view window. Because only a portion of thetiles are actually decompressed, decompression units in accordance withthe present invention requires less processing time than conventionaldecompression units.

After decompression of the subset of relevant tiles, an environmentdisplay system uses the newly formed decompressed environment map totexture map the view window. Specifically, the environment displaysystem uses a texture projection having polygonal curved surfaces as anobject to be texturized using the decompressed environment map. In someembodiments of the present invention, the polygonal curved surfaces aretriangularized to take advantage of conventional hardware renderingunits. By using an efficient texture projection with tiled compressionand partial decompression, the environment maps created by embodimentsof the present invention are ideally suited for immersive videoapplications.

The present invention will be more fully understood in view of thefollowing description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a three-dimensional representation of a user and anenvironment.

FIG. 2 is a three-dimensional representation for texture mapping aspherical environment on a cube.

FIG. 3 is an illustration of a cube face divided into facets.

FIG. 4 is a two-dimensional representation for texture mapping a circleonto a square.

FIGS. 5(a)-5(d) are three-dimensional representations of a sphere withpolygonal curved surfaces as facets.

FIGS. 6(a)-6(b) are illustrations of the division of triangular curves.

FIGS. 7(a)-7(c) are three-dimensional representations of a sphere withpolygonal curved surfaces as facets.

FIGS. 8(a)-8(c) are illustrations of the division of triangular curves.

FIG. 9 is a block diagram of an environment capture and display system.

FIGS. 10(a)-10(b) are environment maps in accordance with embodiments ofthe present invention.

FIGS. 11(a)-11(b) are illustrations of the triangularization of apentagonal curved surface.

FIGS. 12(a)-12(b) are illustrations of the triangularization of atetragonal curved surface.

FIG. 13 is a block diagram of a texture projection unit in accordancewith one embodiment of the present invention.

FIG. 14 is a block diagram of environment capture and display system1400 including compression and decompression units in accordance withone embodiment of the present invention.

FIG. 15 is a block diagram of a compression unit in accordance with oneembodiment of the present invention.

FIGS. 16(a)-(b) are diagrams of images (e.g., environment maps) dividedinto tiles in accordance with one embodiment of the present invention.

FIG. 17 is an illustration of a compressed image in accordance with oneembodiment of the present invention.

FIGS. 18(a)-18(b) are illustrations of compressed images in accordancewith two embodiments of the present invention.

FIG. 19 is a block diagram of a decompression unit in accordance withone embodiment of the present invention.

FIG. 20 illustrates a vertex classification scheme in accordance withone embodiment of the present invention.

FIGS. 21(a)-21(d) illustrates a tile selection scheme in accordance withone embodiment of the present invention.

DETAILED DESCRIPTION

As explained above, environment mapping typically represents theenvironment around a user using a sphere. Specifically, the user's viewof the environment is represented as a texture map on the inside surfaceof the sphere. The environment is texture mapped onto the insidesurfaces of a solid. Typically, a texture projection is formed bydividing the inner surfaces of the solid into polygonal facets. However,as explained above, conventional texture projections are inefficient ascompared to an ideal texture projection. Furthermore, ideal textureprojections are impractical because ideal texture projections can useonly a limited number of facets.

The present invention provides texture projections which provide greaterefficiency than conventional texture projections by using facets thatnearly satisfy the criteria of the facets of an ideal textureprojection. Specifically, the facets used in the present invention arepolygonal curved surfaces rather than polygons. Polygonal curvedsurfaces can be thought of as polygons wrapped onto a base curvedsurface, e.g. a sphere, a cylinder, or a cone. Specifically, an N-sidedpolygonal curved surface has N consecutive sides (S[0]-S[N−1]) whichjoin N consecutive vertices (V[0]-V[N−1]), each of which is located onthe base curved surface. Specifically, each side S[j] joins vertex V[j]to vertex V[j+1 Mod N], where j is an integer from 0 to N−1. The sidesof a polygonal curved surface follow the shortest path along the basecurved surface between the consecutive vertices. The surface of thepolygonal curved surface is congruent to the surface of the base curvedsurface between the sides of the polygonal curved surface. Forconvenience, standard mathematical prefixes are used herein to describespecific polygonal curved surfaces. For example, a tetragonal curvedsurface has four vertices and four sides. Similarly, a pentagonal curvedsurface has five vertices and five sides. However, a polygonal curvedsurface having 3 sides and 3 vertices is referred to as a triangularcurved surface in the same manner as a 3 sided polygon is referred to asa triangle rather than a “trigon.”

In accordance to one embodiment of the present invention, a textureprojection uses polygonal curved surfaces as facets to surround a user.The vertices of the polygonal curved surfaces can be derived recursivelyfrom a plurality of initial polygonal curved surfaces. The initialpolygonal curved surfaces should encompass the desired environment ofviewer 105 (see FIG. 1). Then, each initial polygonal curved surface issubdivided into additional polygonal curved surfaces. The polygonalcurved surfaces formed by dividing the initial polygonal curved surfacesare referred to as second-generation polygonal curved surfaces. Then,each second-generation polygonal curved surface is divided intoadditional polygonal curved surfaces to form a plurality ofthird-generation polygonal curved surfaces. The process of recursivelydividing polygonal curved surfaces into more polygonal curved surfacescontinues until a desired number of facets is reached. The polygonalcurved surfaces forming the facets of the texture projection arereferred to as last-generation polygonal curved surfaces. As the numberof facets increases, the facets become smaller and more planar and thuscan be texture mapped using conventional techniques. For clarity,initial polygonal curved surfaces are referred to by a reference numberR. As polygonal curved surface R is divided into a plurality ofsecond-generation polygonal curved surfaces, each second-generationpolygonal curved surface is referred to in the form R(x) where x is aninteger. Additional parenthesis and indices are added as each newgeneration of polygonal curved surface is formed.

Division of a polygonal curved surface into a plurality ofnext-generation polygonal curved surfaces can be performed in many ways.In accordance with one embodiment of the present invention, a polygonalcurved surface is divided into a plurality next-generation polygonalcurved surfaces so that adjacent polygonal curved surfaces share acommon side and two common vertices. FIGS. 5(a)-5(e) illustrate atexture projection recursively derived in accordance with one embodimentof the present invention. Specifically.in FIG. 5(a), six initialtetragonal curved surfaces 510, 520, 530, 540, 550, and 560 (notvisible) are formed around a spherical base curved surface equivalent tosphere 110. Table 1 (below) provides the vertices of the initialtetragonal curved surfaces of FIG. 5(a) assuming the polygonal curvedsurfaces are centered on the origin of a 3-D space in which the positivex-coordinate points towards the right side of the page, the positivey-coordinate points toward the top of the page, and the positivez-coordinate points into the page. In Table 1, “b” is equal to thesquare root of one-third (approximately 0.57735).

TABLE 1 Curve Vertex 1 Vertex 2 Vertex 3 Vertex 4 510 (−b, −b, −b) (b,−b, −b) (b, b, −b) (−b, b, −b) 520 (−b, b, b) (b, b, b) (b, b, −b) (−b,b, −b) 530 (b, −b, b) (b, −b, −b) (b, b, −b) (b, b, b) 540 (−b, −b, −b)(b, −b, −b) (b, −b, b) (−b, −b, b) 550 (−b, −b, −b) (−b, −b, b) (−b, b,b) (−b, b, −b) 560 (−b, −b, b) (b, −b, b) (b, b, b) (−b, b, b)

Because initial tetragonal curved surfaces 510-560 have the same areaand shape, only division of initial tetragonal curved surface 510,having sides 511, 512, 513 and 514, is shown and explained in detail.Specifically, in FIG. 5(b), initial tetragonal curved surface 510 isdivided into two second-generation tetragonal curved surfaces 510(1) and510(2). Shared side 515 of second-generation tetragonal curved surfaces510(1) and 510(2) is formed by connecting the midpoints of two oppositesides of tetragonal curved surface 510. Specifically, in FIG. 5(b)shared side 515 connects the midpoints of side 511 and side 513. Thus,each second-generation tetragonal curved surface has a first vertex atthe midpoint of a first side of the initial tetragonal curved surface, asecond vertex at the midpoint of a second side of the initial tetragonalcurved surface, a third vertex equal to a first vertex of the initialtetragonal curved surface, and a fourth vertex equal to a second vertexof the initial tetragonal curved surface. In FIG. 5(c) second-generationtetragonal curved surface 510(1) is divided into two third-generationtetragonal curved surfaces 510(1)(1) and 510(1)(2) by a shared side 516joining the midpoints of side 514 and shared side 515. Similarly,second-generation tetragonal curved surface 510(2) is divided into twothird-generation tetragonal curved surfaces 510(2)(1) and 510(2)(2) by ashared side 517 joining the midpoints of side 512 and shared side 515.Table 2 provides the vertices of third-generation tetragonal curvedsurfaces 510(1)(1), 510(1)(2), 510(2)(1), and 510(2)(2) using the samecoordinate system as Table 1. In Table 2 “b” is equal to the square rootof one-third and “c” is equal to the square root of one-half.

TABLE 2 Curve Vertex 1 Vertex 2 Vertex 3 Vertex 4 510 (1) (1) (0, c, −c)(−b, b, −b) (−c, −0, −c) (0, 0, −1) 510 (2) (1) (c, 0, −c) (0, 0, −1)(0, −c, −c) (b, −b, −b) 510 (1) (2) (0, 0, −1) (−c, 0, −c) (−b, −b, −b)(0, −c, −c) 510 (2) (2) (b, b, −b) (0, c, −c) (0, 0, −1) (c, 0, −c)

As illustrated in FIG. 5(d), third-generation tetragonal curved surfaces510(1)(1), 510(2)(1), 510(1)(2), and 510(2)(2) can be further dividedinto fourth generation tetragonal curved surfaces, 510(1)(1)(1) and510(1)(1)(2), 510(2)(1)(1) and 510(2)(1)(2), 510(1)(2)(1) and510(1)(2)(2), 510(2)(2)(1) and 510(2)(2)(2), respectively, by connectingthe midpoints of opposite sides. Additional generations of polygonalcurved surfaces can be derived similarly until a desired number offacets is reached. One benefit of connecting midpoints of opposite sidesis that each polygonal curved surface shares common sides withneighboring polygonal curved surfaces. Vertex coordinates for additionalgenerations of polygonal curved surfaces can be obtained using thesoftware program included in the computer program listing appendix.

FIGS. 6(a)-6(b) illustrate an embodiment of the present invention usingtriangular curved surfaces. Triangular curved surfaces of the same areaand shape can be used to form a sphere in many ways. Specifically, 4, 8,or 20 triangular curves can be used to form a sphere corresponding to atetrahedron, octahedron, and icosahedron, respectively. For brevity andclarity, FIGS. 6(a)-6(b) show the division of a single initialtriangular curved surface 610 into four second generation triangularcurved surfaces 610(1), 610(2), 610(3), and 610(4). Specifically, asillustrated in FIG. 6(b), the sides second generation triangular curvedsurface 610(2) is formed by connecting the midpoints of the sides oftriangular curved surface 610. Additional generations of triangularcurved surfaces can be generated similarly.

As explained above, one condition for an ideal texture projection isthat each facet represents an equal amount of area on sphere 110. Thus,in accordance with another embodiment of the present invention, divisionof a polygonal curved surface creates a plurality of next-generationpolygonal curved surfaces having equal areas. FIGS. 7(a)-7(c) illustratean embodiment of the present invention for deriving a texture projectionusing tetragonal curved surfaces having equal areas. As illustrated inFIG. 7(a), six initial tetragonal curved surfaces 710, 720, 730, 740,750 and 760 (not visible) are formed around a spherical base curveequivalent to sphere 110. Initial tetragonal curved surfaces 710-760 areequivalent to initial tetragonal curved surfaces 510-560 of FIG. 5(a)and Table 1. Because initial tetragonal curved surfaces 710-760 are thesame area and shape, only division of initial tetragonal curved surface710 is shown and explained in detail. In FIG. 7(b), initial tetragonalcurved surface 710 is divided into two second-generation tetragonalcurved surfaces 710(1) and 710(2) have the same area. Specifically, twoopposite sides (sides 710_1 and 710_3) of initial tetragonal curvedsurface 710 are selected. In accordance with one embodiment of thepresent invention, shared side 725 of tetragonal curved surface 710(1)and 710(2) is defined by placing a first vertex of shared side 725 atthe coordinates of a vertex of side 710_1 and a second vertex of sharedside 725 at the coordinates of a vertex of side 710_3. The first andsecond vertices are shifted along sides 710_1 and 710_3, respectively,until second-generation tetragonal curved surfaces 710(1) and 710(2)have the same area. The speed at which the vertices are shifted alongsides 710_1 and 710_3 is directly proportional to the length of theside. Thus, for example, if side 710_1 is twice as long as side 710_2,the first vertex of shared side 725 is shifted twice as quickly as thesecond vertex of shared side 725.

As shown in FIG. 7(c), second-generation tetragonal curved surfaces710(1) is then subdivided into third-generation tetragonal curvedsurfaces 710(1)(1) and 710(1)(2) by a shared side 735. Shared side 735is selected by spanning shared side 725 and side 710_3 to causethird-generation tetragonal curved surfaces 710(1)(1) and 710(1)(2) tohave the same area. Similarly, second-generation tetragonal curvedsurface 710(2) is divided by shared side 745 to form third generationtetragonal curved surfaces 710(2)(1) and 710(2)(2) having the same area.Additional generations of tetragonal curved surfaces are formedsimilarly until a desired number of facets is reached. The vertices forthird-generation tetragonal curved surfaces 710(1)(1), 710(1)(2),710(2)(2), and 710(2)(1) are the same as for second-generationtetragonal curved surfaces 510(1)(1), 510(2)(1), 510(2)(2), and 510(1)(2), respectively. However division of third-generation tetragonalcurved surface 710(1)(1), as described with the method of FIGS.7(a)-7(c), will result in polygonal curved surfaces that are notequivalent to the division of third-generation tetragonal curved surface510(1)(1), as illustrated in FIG. 5(d). Vertex coordinates foradditional generations of polygonal curved surfaces can be obtainedusing the software program included in the computer program listingappendix.

In accordance with some embodiments of the present invention, multiplemethods to divide the polygonal curved surfaces may be used to form asingle texture projection. Furthermore, polygonal curved surfaces may bedivided into next-generation polygonal curved surfaces having adifferent number of sides. For example, as illustrated in FIGS. 8(a),8(b) and 8(c), one embodiment of the present invention divides aninitial triangular curved surface 810 into three second-generationtetragonal curved surfaces 810(1), 810(2), and 810(3). Initialtriangular curved surface 810 is divided by forming shared sides 811,812, and 813 from the midpoints of the sides of initial triangular curve810 to the center of initial triangular curved surface 810. As shown inFIG. 8(c), second-generation tetragonal curved surfaces 810(1), 810(2),and 810(3) can then be subdivided into third-generation tetragonalcurved surfaces 810(1)(1)-810(1)(4), 810(2)(1)-810(2)(4), and810(3)(1)-810(3)(4), respectively, using the method illustrated in FIGS.5(a)-5(c) or FIGS. 7(a)-7(c).

FIG. 9 shows an environment capture and display system 900 having anenvironment map creation system 910, a data transport system 920, and anenvironment display system 930. Environment map creation system 910creates an environment map 940 for the environment of viewer 105, i.e.,the inner surface of sphere 110 (FIG. 1). Specifically an environmentcapture/generation unit 915 captures or generates one or more images torepresent the environment of viewer 105. For example, in some systemsenvironment capture/generation unit 915 contains a camera system whichcan capture the entire environment of viewer 105. Some embodiments ofenvironment capture/generation unit 915 use multiple cameras to takemultiple pictures at various angles centered around viewer 105. Amultiple camera system typically provides very high resolution images,but also includes redundant data due to overlapping views from thecameras. In other embodiments, environment capture/generation unit 915generates an artificial environment for viewer 105. The generatedenvironment can be stored as a single image or multiple images atvarying resolution.

Next environment data is passed to environment map rendering unit 917.Environment map rendering unit 917 also receives a texture projection914 from texture projection generation unit 912. The number of facets oftexture projection 914 is usually chosen to equal the desired resolutionof environment map 940. Conceptually, environment map rendering unit 917forms an environmental surface surrounding viewer 105 from the one ormore images supplied by environment capture/generation unit 915.Conventional image stitching techniques can be used to join multipleimages. For example, if environment capture/generation unit 915 is asix-camera system, environment map rendering unit 917 conceptually formsa cube (such as cube 220 in FIG. 2) around viewer 105 using six imagesfrom environment capture/generation unit 915.

Then, environment map rendering unit 917 determines the area on theenvironmental surface corresponding to each of the facets in textureprojection 914. Conceptually, the corresponding area is determined byforming a solid angle encompassing the texture and projecting the solidangle onto the environmental surface. The corresponding area is the areaof the environmental surface intersecting the solid angle. As statedabove, typically the number of facets is selected to equal the desiredresolution of the environment map. Thus, each facet corresponds to onetexel on the environment map. Accordingly, the facet has a single colorthat is determined by averaging the colors of the pixels in thecorresponding area on the environmental surface. However, in someembodiments of the present invention, a facet corresponds to multiplepixels on the environment map. For these embodiment the facet hasmultiple colors based on the corresponding area of the environmentalsurface.

In actual implementation, environment map rendering unit 917 candetermine the image and area in that image which corresponds to eachfacet based on the camera system configuration. Some facets maycorrespond to multiple images, e.g., a facet which projects onto theintersection of one or more images. The color for these facets caneither be determined by using only one image, or by averaging theappropriate area of each image.

Once the color or colors of each facet is determined, environment maprendering unit 917 generates environment map 940 by treating eachinitial polygonal curved surface of the texture projection as atwo-dimensional polygonal image. Each facet within an initial polygonalcurved surface becomes a texel in the corresponding two-dimensionalpolygonal image. The two-dimensional polygonal images are thenconcatenated together to form environment map 940. For example, FIG.10(a) shows an environment map 1010 that could be formed using a textureprojection based on FIGS. 5(a)-5(d). Specifically, initial tetragonalcurved surfaces 510, 520, 530, 540, 550, and 560, are converted intotwo-dimensional tetragonal images 1011, 1012, 1013, 1014, 1015, and1016, respectively. Two-dimensional tetragonal images 1011-1016 areconcatenated together to form environmental map 1010. In someembodiments of the present invention, square environmental maps aredesired. For these embodiments, the two-dimensional tetragonal imagesmay have different resolutions. For example, in FIG. 10(b), anenvironmental map 1020 having a resolution of 1024×1024 is formed bytwo-dimensional tetragonal images 1021, 1022, 1023, 1024, 1025, and1026. Two-dimensional tetragonal images 1021 and 1025 have a resolutionof 512×512. Two-dimensional tetragonal images 1022-1025 have aresolution of 512×256. Two-dimensional tetragonal images 1022-1025 maybe formed by forming a 512×512 image and reducing it to 512×256 usingconventional techniques. Alternatively, initial polygonal curvedsurfaces corresponding to two-dimensional tetragonal images 1022-1025may have 512×256 facets.

After environment map creation system 910 (FIG. 9) creates environmentmap 940, environment map 940 is transported to environment displaysystem 930 by data transport system 920. In some embodiments of thepresent invention, data transport system 920 is a data channel, such asa local area network, a telephone line, or the internet. In otherembodiments of the present invention, data transport system 920 is astorage medium, such as a CD-ROM, a DVD, or a data tape.

Environment display system 930 receives environment map 940 from datatransport system 920 and displays the environment as a textureprojection on a display 955. Specifically, environment display system930 includes a data storage unit 935, a texture projection generationunit 932, an optional triangularization unit 938, a texture renderingunit 937, display 955, a user input device 952, and a view windowdetermination unit 953. Environment map 940 is stored in data storageunit 935. In some embodiments of the present invention data storage unit935 is a computer memory system or a data storage system (e.g., a diskdrive) of a computer system. Display unit 955 is typically a computermonitor, a head-mounted display, or a television set. User interfacedevice 952 can be for example, a joystick, a mouse, a track ball, ahead-tracking device, or a keyboard. View window determination unit 953provides a view window 955 which indicates the area that is visible toviewer 105 (FIG. 1). Generally, view window determination unit 953determines view window 954 based on user input from user input device952.

Texture projection generation unit 932 creates a texture projection 934as described above. Usually, texture projection 934 uses the same basecurved surface and the same set of initial polygonal curved surfaces asused by texture projection 914. However, the number of facets in textureprojection 934 need not equal the number of facets in texture projection914. Texture rendering unit 937 texture maps environment map 940 in datastorage unit 935 onto a visible portion of texture projection 934.Specifically, texture rendering unit 937 aligns the initial polygonalcurved surfaces of the texture projection from texture projection unit932 with the two-dimensional polygonal images of the environment map.Then, the color for each vertex of a facet is read from the appropriatetwo-dimensional polygonal image of the environment map. If a facetcontains multiple pixels, the color for the non-vertex pixels can beinterpolated can be retrieved from the texture map by interpolating thetexture coordinates from the vertex coordinates. This process isrepeated for each facet in the visible portion of texture projection934. The visible portion of texture projection 934 is typicallydetermined by view window 954. Conventional texture mapping and lineclipping techniques are used by texture rendering unit 937 to create theimage on display 955 based on view window 954, environment map 940, andtexture projection 934.

Some embodiments of texture rendering unit 937 are optimized fortexturing triangles. Thus, in some embodiments of the present invention,texture projection 934 is triangularized by triangularization unit 938for texture rendering unit 937. Triangularization of texture projection934 involves converting the facets of texture projection 934 frompolygonal curved surfaces into triangles. For triangular curvedsurfaces, triangularization is accomplished by using the vertices ofeach facet as a vertex of a triangle rather than a vertex of atriangular curved surface.

FIGS. 11(a)-11(b) illustrate a method to triangularize a pentagonalcurved surface in accordance with one embodiment of the presentinvention. However, the method of FIGS. 11(a)-11(b) can easily beadapted to triangularize any polygonal curved surface. FIG. 11(a) showsa pentagonal curved surface 1110 having vertices 1111-1115. Asillustrated in FIG. 11(b), a triangularization vertex 1116 is selectedon pentagonal curved surface 1110. Usually, triangularization vertex1116 is at the center of pentagonal curve 1110. Each pair of adjacentvertices of pentagonal curve 1110 and triangularization vertex 1116together form the vertices of a triangle. Thus, pentagonal curvedsurface 1110 is triangularized into triangle 1151 having vertices 1115,1111, and 1116; triangle 1121 is formed having vertices 1112, 1111, and1116; triangle 1132 having vertices 1113, 1112, and 1116; triangle. 1143having vertices 1114, 1113, and 1116; and triangle 1154 having vertices1115, 1114, and 1116.

As illustrated in FIGS. 12(a)-12(b), a tetragonal curved surface 1210having vertices 1211, 1212, 1213, and 1214 can be triangularized into atriangle 1240 having vertices 1211, 1213, and 1214, and a triangle 1220having vertices 1211, 1213, and 1212. The triangularization methodillustrated in FIGS. 12(a)-12(b) would be equivalent to thetriangularization method of FIGS. 11(a)-11(b) if the triangularizationvertex is selected to be equal to one of the vertices of the polygonalcurved surface.

In some embodiments of the present invention, dedicated hardwareimplementations of texture rendering unit 937 and texture projectiongeneration unit 932 are used. However, most embodiments of the presentinvention use a processor to execute software implementations of texturerendering unit 937 and texture projection generation unit 932. Althoughsome embodiments may use a combination of hardware and softwareimplementations.

FIG. 13 is a block diagram of one embodiment of texture generationprojection generation unit 912, which generates texture projection 914.Specifically, the embodiment of FIG. 13 includes a facet storage unit1310, an initial polygonal curved surface generator 1320, and apolygonal curved surface divider 1330. Initial polygonal curved surfacegenerator 1320 receives initial data 1325 for generating textureprojection 914. Initial data 1325 may include information such as theshape of the initial polygonal curved surfaces, the base curved surfaceto be used, and the number of initial polygonal curved surfaces. Frominitial data 1325, initial polygonal curved surface generator 1320generates the initial polygonal curved surfaces for texture projection914 and stores the initial polygonal curved surfaces in facet storageunit 1310. Facet storage unit 1310 is typically a random access memory(RAM) device. For example, in one embodiment of texture projectiongeneration unit 912, facet storage unit 1310 is part of the memorysystem of a general purpose computer.

After the initial polygonal curved surfaces are generated, polygonalcurved surface divider 1330 divides the initial polygonal curvedsurfaces into a plurality of second-generation polygonal curvedsurfaces. Polygonal curved surface divider 1330 is controlled bydivision data 1335, which may include information such as the divisionmethod for creating the next-generation polygonal curved surfaces, thenumber of generations, or the number of facets. Polygonal curved surfacedivider 1330 recursively divides each generation of polygonal curvedsurfaces into a group of next-generation polygonal curved surfaces.Specifically, polygonal curved surface divider 1330 retrieves eachZ-generation polygonal curved surface from facet storage unit 1310,divides the retrieved Z-generation polygonal curved surface into aplurality of Z+1-generation polygonal curved surfaces, and stores theZ+1-generation polygonal curved surfaces back into facet storage unit1310. After polygonal curved surface divider 1330 finishes dividing thepolygonal curved surfaces, the last generation of polygonal curvedsurfaces in facet storage unit 1310 forms texture projection 914.

As explained above, immersive videos, which are composed of hundreds oreven thousands of environment maps, are a natural extension ofenvironment mapping. However, the large amount of data required forimmersive videos may be beyond the processing capabilities of mostcomputer systems. Conventional compression techniques have been used toreduce the amount of data required for immersive videos. However, theprocessing requirements to decompress the environment maps as well asdisplaying the proper portions of the environment map may be beyond theprocessing power of most environmental display systems.

Accordingly, some embodiments of the present invention use novelcompression and decompression units to compress the environment mapswithout requiring excessive processing for decompression. FIG. 14 showsan embodiment of an environment capture and display system 1400, whichcan be used for creating and displaying immersive videos. Environmentcapture and display system 1400 is similar to environment capture anddisplay system 900 (FIG. 9), thus the same reference numerals are usedto describe similar elements. Furthermore, for brevity descriptions ofthe similar elements are not repeated. Environment capture and displaysystem 1400 includes a compression unit 1410 and a decompression unit1420. Compression unit 1410 receives environment map 940 fromenvironment map creation system 910 and creates a compressed environmentmap 1430 to be transported by data transport system 920. Decompressionunit 1420 is part of environment display system 930 and is configured topartially decompress compressed environment map 1430 for texturerendering unit 937. Specifically, compression unit 1410 compressesenvironment map 940 so that decompression unit 1420 can decompressspecific parts of compressed environment map 1430, rather than requiringdecompression of compressed environment map 1430 in its entirety.Decompression unit 1420 receives view window 954, identifies theportions of compressed environment map 1430 that are needed by texturerendering unit 937, and decompresses the needed portions. In someembodiments of the present invention, decompression unit 1420 usestexture projection 934 to convert the coordinate systems of view window954 to the coordinate system of compressed environment map 1430 or viceversa. Because only part of compressed environment map 1430 isdecompressed, decompression unit 1420 requires far less processing timethan conventional decompression units. Decompression unit 1420 isexplained in further detail below with respect to FIG. 19.

FIG. 15 is a block diagram of compression unit 1410 in accordance withan embodiment of the present invention. The embodiment of FIG. 15includes a tiling unit 1510, a tile compressor 1520, a header formationunit 1530, and a compressed image collation unit 1540. Tiling unit 1510receives an image 1505, which can be, for example, environment map 940(FIG. 14), and configuration information 1506. Configuration information1506 provides information such as the tile size or sizes, the verticesof specific tiles, and/or other parameters for tiling unit 1510. Asillustrated in FIG. 16(a), tiling unit 1510 divides image 1505 into aplurality of tiles such as tile 1610, 1620, and 1630. Tiles areillustrated using dashed lines in FIG. 16(a) and 16(b). Generally,tiling unit 1510 uses rectangular tiles of the same shape and area.However, some embodiments may use other shapes having different sizesand areas. Some embodiments of tiling unit 1510 are pre-configured for aspecific tiling pattern, and would not require configuration information1506. As explained above, environment maps are typically formed byconcatenating a plurality of the two-dimensional polygonal images. Toease the burden on decompression unit 1420, tiling unit 1510 generallylimits a tile to be contained in only one of the two-dimensionalpolygonal images. Specifically, as illustrated in FIG. 16(b),environment map 1020 of FIG. 10(b) is tiled so that no tile crosses aborder of two-dimensional polygonal images 1021-1026.

Once image 1505 has been tiled, tile compressor 1520 compresses eachtile individually. Since each tile can be considered as a separatetwo-dimensional image, tile compressor 1520 can use conventional imagecompression methods, such as JPEG, run-length encoding, and GIF. Tilecompressor 1520 provides each compressed tile to compressed imagecollation unit 1540, and provides the size of each compressed tile toheader formation unit 1530.

Header formation unit 1530 creates a header 1710 (FIG. 17) for acompressed image 1545. As illustrated in FIG. 17, compressed image 1545is a binary string of data formed by header 1710 followed by N (thenumber of tiles used by tiling unit 1510) compressed tiles 1545_1,1545_2, . . . 1545_N. In some embodiments of the invention, header 1710contains a tile descriptor for each compressed tile. Each tiledescriptor may contain information, such as the size of thecorresponding compressed tile (typically given in bytes), the shape ofthe corresponding tile in image 1505, and the vertices of thecorresponding tile in image 1505. For embodiments of compression unit1410 that are pre-configured for a specific tile size, the tiledescriptor in header 1710 might only contain the sizes of the compressedtiles. Alternatively, as illustrated in FIG. 18(a), a compressedenvironment map 1800 contains N compressed tiles 1800_1, 1800_2, . . .1800_N, preceded by a header 1810 which contains N offset values 1810_1,1810_2, . . . 1810_N. Each offset value 1810_x indicates the location ofcompressed tile 1800_x in compressed image 1800. Each offset value1810_x can be computed by adding the size of compressed tile 1800_(x−1)to offset value 1810_(x−1), where x is an integer between 2 and N,inclusive. Offset value 1810_1 is equal to the size of header 1810 whichis equal to N times the number of bytes used per offset value. Thus,offset values of header 1810 are also considered as tile descriptors.

FIG. 18(b) illustrates a compressed environment map 1830 in accordancewith one embodiment of the invention. Compressed environment map 1830includes a header 1840 followed by 64 compressed tiles 1860_1-1860_64,followed by a four-byte format number 1870. Format number 1870 can beany four-byte number as agreed upon between compression unit 1410 anddecompression unit 1420. Four-byte format number 1870 allowsdecompression unit 1420 to insure that compressed environment map 1830is in the proper format. Header 1840 includes map length 1841 as a fourbyte number, a second format number 1842, 64 four-byte offset values1850_1-1850_64 corresponding to compressed tiles 1860_1-1860_64,respectively, and compression information 1843. Specifically, theembodiment of FIG. 18(b) uses JPEG compression using the same JPEGcoefficient table to form each compressed tile 1860_1-1860_64. Ratherthan storing a copy of the JPEG coefficient table with each compressedtile, the JPEG coefficient table is stored in compression information1843.

As explained above with respect to FIG. 14, decompression unit 1420decompresses only a portion of compressed environment map 1430 based onview window 954. Generally, only compressed tiles that contain relevantdata, i.e., texels needed to create the environment within view window954 need to be decompressed. However, determination of exactly whichcompressed tiles contain relevant data may be more processing intensivethan decompressing a few irrelevant tiles, i.e., tiles that do notcontain relevant data. Thus, some embodiments of the present inventionselect and decompress a subset of the compressed tiles, where the subsetmay contain irrelevant tiles. However, the subset of compressed tilesfor decompression does include all compressed tiles having relevantdata.

FIG. 19 is a block diagram of one embodiment of decompression unit 1420which can be used with the environment maps as described above. Theembodiment of FIG. 19 includes a view frustum calculator 1910, anoptional coordinate conversion unit 1920, a tile vertex classifier 1930,a tile selector 1940, and a tile decompressor 1950. View frustumcalculation unit 1910 receives view window 954 and calculates the normalvectors of a view frustum encompassing view window 954. A view frustumis the solid angle projection from viewer 105 (typically at the origin)which encompasses view window 954. Generally, view window 954 isrectangular, thus the view frustum for view window 954 would resemble afour sided pyramid and have four normal vectors, i.e., one for each sideof the view frustum. A view frustum normal vector points perpendicularto the plane containing a side of the view frustum. The embodimentsdescribed herein use view frustum normal vectors that point into theview frustum. Other embodiments may use view frustum normal vectors thatpoint out of the view frustum. If view window 954 is not rectangular, arectangular view frustum can be created by using a rectangular secondaryview window that encompasses view window 954. However, additionalirrelevant tiles may be decompressed by using the rectangular secondaryview window.

The view frustum normal vectors are provided to tile vertex classifier1930. Compressed image 1905, which can be for example compressedenvironment map 1430 (FIG. 14) is also provided to tile vertexclassifier 1930. Generally, the coordinate system of the view window 954is the same as the coordinate system of compressed image 1905. However,in some embodiments the coordinate systems differ and coordinateconversion unit 1920 converts the coordinates of the vertices of thecompressed tiles to match the coordinate system of view window 954.

Tile vertex classifier 1930 uses the view frustum normal vectors toclassify the vertex of each compressed tile to determine whether thevertex is above, below, left or right of the view frustum. Above, below,left and right are relative to the view frustum, rather than the vieweror some other fixed object. Tile vertex classifier 1930 can extract thevertices from the header of compressed image 1905. Alternatively, tilevertex classifier 1930 may use a predefined set of tile vertices, whichis also used by tiling unit 1510 (FIG. 15). The relationship of a vertexwith the view frustum is computed using the inner product (or dotproduct) of the vertex with the view frustum normal vectors. Forexample, if the inner product of a vertex with the right side viewfrustum normal vector is less than zero, then the vertex is to the rightof the view frustum. Similarly, if the inner product of a vertex withthe left side view frustum normal vector is less than zero, then thevertex is to the left of the view frustum. Table 4 below provides pseudocode for one implementation of tile vertex classifier 1930. Using viewfrustum normal vectors with vertices on the opposite side (through theorigin of the coordinate system) of the view window may cause strangeseeming results. For example, a vertex on the opposite side of viewwindow 954 may be classified as left of, right of, above, and below theview frustum. However, these abnormal vertex classifications can easilybe avoided by tile selector 1940. For example, titles on the oppositeside of view window 954 can be eliminated by using a view axis vector,which points from the origin to the center of view window 954.Specifically, if the inner product of the view axis vector with each ofthe vertices of a tile is less than zero, the tile is on the oppositeside of view window 954 and can be ignored.

In one embodiment of tile vertex classifier, a 4-bit coding scheme isused to classify each vertex, i.e., each vertex is given a 4 bitbit-code classification. Specifically, bit 3 indicates whether a vertexis above the view frustum, bit 2 indicates whether a vertex is below theview frustum, bit 1 indicates whether a vertex is to the right of theview frustum, and bit 0 indicates whether the vertex is to the left ofthe view frustum. For clarity, the bit coding scheme is described hereinusing 1 as the true state and 0 as the false state. Thus, if a vertex isleft of the view frustum bit 0 of the bit code for the vertex is setto 1. If a vertex is both above and to the left of the view frustum thenboth bit 3 and bit 0 of the bit code is set to 1. If a vertex is in theview frustum, i.e., it is not to the left, not to the right, not above,and not below the view frustum, the vertex would have a bit code of 000b(the “b” as used herein indicates a binary number). FIG. 20 illustratesthe bit coding scheme. For clarity, FIG. 20 shows only a two dimensionalslice of the view frustum. Specifically, FIG. 20 shows a view frustuminterior 2080 and various vertices 2000, 2001, 2002, 2004, 2005, 2006,2008, 2009, and 2010. Table 3 provides the region attribute andcorresponding bit code for the vertices of FIG. 20.

TABLE 3 Attribute with respect to Vertex View Frustum Bit Code 2000Inside 0000b 2001 Left 0001b 2002 Right 0010b 2004 Below 0100b 2005Below and Left 0101b 2006 Below and Right 0110b 2008 Above 1000b 2009Above and Left 1001b 2010 Above and Right 1010b

Table 4 contains pseudo code for an embodiment of tile vertex classifierfor use with the environment mapping system described above.Furthermore, the embodiment of Table 4 uses the bit-code schemedescribed above and illustrated by FIG. 20 and Table 3.

TABLE 4 Variable Definition: T_tot = number of tiles T_Vert = number ofvertices per tile V (x) (y) = Vertex number y of tile number x V_BC (x)(y) = Bit code for Vertex number y of tile number x NV (z) = Normalvector number z where z=0 is left, z=1 is right, z=2 is below, and z=3is above Code: for x = 1 to T_tot {x cycles through the tiles} for y = 1to T_Vert {y cycles through the vertices on each tile} V_BC (x) (y)=0000b for z = 1 to 4 {z cycles through the view frustum normal vectors}if inner_product (V (x) (y), NV (z)) < 0 then V_BC (x) (y) =V_BC (x)(y) + 2{circumflex over ( )}z next z next y next x

Decompression unit 1420 may be used with flat two-dimensional images.For example, some applications only display a portion of a largepicture. The displayed portion can be considered a view window. Vertexclassification is performed on a two-dimensional coordinate system withthe origin of the coordinate system in the bottom left corner of theimage. A vertex can be classified by comparing the coordinates of thevertex with the coordinates of the bottom left vertex of the view windowand with the coordinates of the top right vertex of the view window.Table 4 contains pseudo code for an embodiment of tile vertex classifierfor use with flat two-dimensional images.

TABLE 5 Variable Definition: T_tot = number of tiles T_Vert = number ofvertices per tile V (x) (y) = Vertex number y of tile number x X_V (x)(y) = X-coordinate of vertex number y of tile number x Y_V (x) (y) =Y-coordinate of vertex number y of tile number x V_BC (x) (y) = Bit codefor Vertex number y of tile number x X_VW_BL = X−coordinate of thebottom left vertex of the view window Y_VW_BL = Y−coordinate of thebottom left vertex of the view window X_VW_TR = X−coordinate of the topright vertex of the view window Y_VW_TR = Y−coordinate of the top rightvertex of the view window Code: for x = 1 to T_tot {x cycles through thetiles} for y = 1 to T_Vert {y cycles through the vertices on each tile}V_BC (x) (y) =0000b if (X_V (x) (y) <X_VW_BL) then V_BC (x) (y) =V_BC(x) (y) + 0001b if (X_V (x) (y) >X_VW_TR) then V_BC (x) (y) =V_BC (x)(y) + 0010b if (Y_V (x) (y) <Y_VW_BL) then V_BC (x) (y) =V_BC (x) (y) +0100b if (Y_V (x) (y) >Y_VW_TR) then V_BC (x) (y) =V_BC (x) (y) + 1000bnext y next x

Tile vertex classifier 1930 provides the vertex classifications to tileselector 1940. Tile selector 1940 then selects a subset of the tiles incompressed image 1905 for decompression unit 1950. The subset ofselected tiles chosen by tile selector 1940 should contain all tileswhich contain relevant data, i.e., visible texels. Inclusion ofirrelevant tiles, i.e., tiles containing only invisible texels, shouldbe minimized. However, the processing time required to eliminate allirrelevant tiles from the subset of selected tiles may be greater thanthe processing time of decompressing a few irrelevant tiles. Thus, manyembodiments of the present invention do not completely eliminateirrelevant tiles from the subset of selected tiles.

FIG. 21(a)-(d) illustrate how one embodiment of tile selector 1940selects a subset of tiles in compressed image 1905. For clarity, FIGS.21(a)-(d) show only a two dimensional slice of the view frustum.Specifically, FIGS. 21(a)-21(d) show a view frustum interior 2110 andvarious tiles 2120, 2130, 2140, 2150, 2160, 2170, 2180, and 2190. A tilecontains visible texels if any part of the tile is within the viewfrustum. A tile can become visible in only three basic situations.First, the tile can be completely within view frustum interior 2110,such as tile 2120. Second, the tile can completely encompass the viewfrustum, such as tiles 2170 (FIG. 21(b)) and 2180 (FIG. 21(c)). Finally,the tile contains relevant data if the tile is only partially within theview frustum, such as tiles 2130, 2140 2150, and 2160. Each of thesethree conditions can be detected separately or a single test may detectmultiple conditions.

For example, if a tile is partially within the view frustum at least oneside of the tile must cross one of the sides of the view frustum.Analysis of the vertex classifications can detect all such tiles.Specifically, for any tile that is partially within the view frustum,the bitwise AND operation of the bit-code (as defined above) of at leastone pair successive vertices must equal 0000b. As illustrated in FIGS.21(a), vertices 2131 and 2132 of tile 2130 have bit codes of 0001b and0000b, respectively. Thus, the bitwise logic AND of the bit codes ofvertices 2131 and 2132 is equal to 0000b. Table 6 provides the bit-codesand the bitwise logic AND of various successive vertices of the tiles inFIG. 21(a). Note that at least one successive pair of each tile that ispartially within the view frustum is equal to 0000b.

TABLE 6 TILE VERTEX Bit-Code Vertex Bit-Code Bitwise Logic AND 2130 21310001b 2132 0000b 0000 2140 2141 0001b 2142 1000b 0000 2150 2151 1000b2152 0000b 0000 2160 2161 1000b 2162 0100b 0000

The bitwise logic AND test between two successive vertices also detectstiles completely within the view frustum, such as tile 2120, because allthe vertices of such tiles is equal to 0000b. Furthermore, the bitwiselogic AND test between two successive vertices also detects certaintypes of tiles which encompass the view frustum, such as tile 2170 (FIG.21(b)). Specifically, if a tile has a first vertex that is only above oronly below the view frustum, and a second successive vertex that is onlyto the left or only to the right of the view frustum, then the bitwiselogic AND of the first and second vertices is equal to 0000b. Forexample, vertices 2172 and 2171 of tile 2170 have bit-codes 1000b and0001b, respectively. Thus, the bitwise logic AND of the bit-codes ofvertices 2172 and 2171 is equal to 0000b. However, the bitwise logic ANDtest also selects irrelevant tiles such as tile 2190 (FIG. 21(d)), whichcontains no visible texels, because vertices 2191 and 2192 have the samebit-codes as vertices 2171 and 2172, respectively. Although inclusion ofirrelevant tiles is inefficient, as explained above, the processing timeof decompressing the irrelevant tile may be less than the processingtime required to eliminate the irrelevant tiles from the subset ofselected tiles.

A specific class of visible tiles is not detected by the bitwise logicAND test. As shown in FIG. 21(c), tile 2180 encompasses the view frustumand is thus partially visible. However, bitwise logic AND of any twosuccessive vertices is not equal to 0000b. Thus, a second test isrequired to detect this class of tiles. For this class of tiles, thebitwise exclusive-OR (XOR) of the bit-code of both pair of oppositevertices results in 1111b. For example, the bit-codes of vertices 2181and 2183 are equal to 1001b and 0110b. Thus, the bitwise XOR of thebit-codes of vertices 2181 and 2183 is equal to 1111b. Similarly, thebit codes of vertices 2182 and 2184 are equal to 1010b and 0101b. Thus,the bitwise XOR of the bit-codes of vertices 2182 and 2184 is equal to1111b. The combination of the bitwise logic AND test with the bitwiseXOR test detects all tiles having visible texels. Table 7 provides thepseudo code for an embodiment of tile selector 1940 using the bitwiselogic AND test and the bitwise XOR test to select a subset of tiles.

TABLE 7 Variable Definition: T_tot = number of tiles T_Vert = number ofvertices per tile V (x) (y) = Vertex number y of tile number x V_BC (x)(y) = Bit code for vertex number y of tile number x T_VIS (x) = Binaryflag indicating tile number x is selected as a member of the subset ofselected tiles. & indicates a bitwise logic AND function # indicates abitwise XOR function == is read as “is equal” Code: for x = 1 to T_tot{x cycles through the tiles} T_VIS (x) =0 for y = 1 to T_Vert {y cyclesthrough the vertices on each tile} if (V (x) (y) & V (x) ((y+1) MODT_Vert) == 0 then T_VIS (x) =1 next y if T_VIS (x) =0 then if (V (x) (1)# V (x) (3)) == 1111b) and (V (x) (2) # V (x) (4)) == 1111b) then T_VIS(x) =1 next x

The subset of selected tiles are sent to tile decompressor 1950, whichdecompresses the selected tiles using the corresponding decompressionmethod to the compression method used by tile compressor 1520 (FIG. 15).The decompressed tiles are sent to decompressed image collation unit1960, which also receives the corresponding vertex coordinates or tilenumber for each decompressed tile. Decompressed image collation unit1960 produces a partially decompressed image 1965, which includes alltiles that contains visible texels.

Thus, texture rendering unit 937 (FIG. 9) receives a decompressedtexture map that contains all the necessary texels to texture map viewwindow 954. Because decompression unit 1420 only needs to partiallydecompress compressed environment map 1430 (FIG. 14), environmentdisplay system 930 can display a high resolution flicker-free immersivevideo on display 955.

In the above-described manner, high-resolution flicker-free immersivevideos are made possible. Specifically, an immersive video system inaccordance with embodiments of the present invention combine anefficient texture projection with a compression scheme, that allowspartial decompression of the environment map. The various embodiments ofthe structures and methods of this invention that are described aboveare illustrative only of the principles of this invention and are notintended to limit the scope of the invention to the particularembodiments described. For example, in view of this disclosure, thoseskilled in the-art can define other polygonal curved surfaces, curvedsurfaces, curve division methods, environment mappings, facets, texels,tile selectors, tile compressors, tile decompressors, compression units,decompression units, tile vertex classifiers, and so forth, and usethese alternative features to create a method or system according to theprinciples of this invention. Thus, the invention is limited only by thefollowing claims.

APPENDIX I Copyright (©) 1999 Enroute Inc. All Rights Reserved FILENAME:subdivide.cpp #include <cstdlib> #include <cmath> #include <limits>#include ″tess.h″ void usage(const char *cmd) { std: :cout << cmd << ″nlevels method\n\n″; std: :cout << ″ This sample program projects theface of a cube onto a\n″; std: :cout << ″co-centered sphere, andrecursively subdivides the\n″; std: :cout << ″projection intoquadrilaterals.\n\n″; std: :cout << ″The program requires two integercommand line arguments: \n″; std: :cout << ″ nlevels number ofsubdivision levels\n″; std: :cout << ″ method 0 for cartesiansubdivision, \n″; std: :cout << ″ 1 for arc subdivision,\n″; std: :cout<< ″ 2 for area subdivision\n″; } main(int argc,char *argv[]) { if (argc != 3 ) { usage (argv [0]); exit (−1); } Quad q( Vector3 (−1.0,−1.0, −1.0), Vector3 ( 1.0, −1.0, −1.0), Vector3 ( 1.0,  1.0, −1.0),Vector3 (−1.0,  1.0, −1.0) Tess: :SubType type; switch ( atoi(argv[2]) ){ case 0: type = Tess: :Cartesian; break; case 1: type = Tess: :Arc;break; case 2: type = Tess: :Area; break; default: usage (argv [0]);exit (−1) } Tess t (q, atoi (argv [1]), type); double minArea = std::numeric_limits<double>: :max(); double maxArea = 0.0; double minRatio =std: :numeric_limits<double>: :max(); double maxRatio = 0.0; doubleareaSum = 0.0; double areaSum2 = 0.0; for ( Tess: :QuadList: :iterator i= t.quad.begin(); i != t.quad.end(); i++ ) // print out the quadvertices std: :cout << *i << ″\n″; const double area = i->sarea();areaSum += area; areaSum2 += area*area; if ( area < minArea ) minArea =area; if ( area > maxArea ) maxArea = area; const double ratio =i->aratio(); if ( ratio < minRatio ) minRatio = ratio; if ( ratio >maxRatio ) maxRatio = ratio; } const double areaAvg =areaSum/static_cast<double>(t.quad.size() ); std: :cout << t.quad.size()<< ″ quads\n″; std: :cout << ″min area ″ << minArea << ″ steradians\n″;std: :cout << ″max area ″ << maxArea << ″ steradians\n″; std: :cout <<″average area ″ << areaAvg << ″\n″; std: :cout << ″area variance ″ <<(areaSum2 − 2.0*areaAvg*areaSum +areaAvg*areaAvg*static_cast<double>(t.quad.size() ) )/static_cast<double>(t.quad.size() − 1) << ″\n″; std: :cout << ″min arcratio ″ << minRatio << ″\n″; std: :cout << ″max arc ratio ″ << maxRatio<< ″\n″; return 0;} FILENAME : tess.h #ifndef _TESS_H_ #define _TESS_H_#include <list> #include ″quad.h″ class Tess { public: enum SubType{Cartesian, // subdivide in the plane Arc, // subdivide by equal arclength Area // subdivide by equal area }; Tess(const Quad& q,intnLevels,SubType type); typedef std: :list<Quad> QuadList; QuadList quad;protected: int nLevels; // number of subdivision levels SubType type; //subdivision type //: recursive subdivision function void divide(constQuad& q,int level); //: planar subdivision void dividePlane(const Quad&q,Quad& sub0,Quad& sub1); //: arc division void divideArc(const Quad&q,Quad& sub0,Quad& sub1); //: area division void divideArea(const Quad&q,Quad& sub0,Quad& sub1); }; #endif _TESS_H_(—) FILENAME: tess.cpp#include <limits> #include ″tess.h″ Tess: :Tess(const Quad& q,int_nLevels,SubType _type) : nLevels(_nLevels), type(_type) { Quad qs = q;if ( type != Cartesian ) { qs.v[0].normalize(); qs.v[1].normalize()qs.v[2].normalize(); qs.v[3].normalize(); } divide(qs,0) } void Tess::divide(const Quad& q,int level) { if ( level == 2*nLevels ) {quad.push_back(q); return; } Quad q0,q1; switch (type) { case Cartesian:dividePlane (q,q0,q1); break; case Arc: divideArc(q,q0,q1); break; caseArea: divideArea(q,q0,q1) break; } q0.rotateVertices ();q1.rotateVertices (); divide(q0,level + 1); divide(q1,level + 1); } voidTess: :dividePlane (const Quad& q,Quad& sub0,Quad& sub1) { Vector3 v01;v01.x = 0.5*(q.v[0].x + q.v[1].x); v01.y = 0.5*(q.v[0].y + q.v[1].y);v01.z = 0.5*(q.v[0].z + q.v[1].z); Vector3 v32; v32.x = 0.5*(q.v[3].x +q.v[2].x); v32.y = 0.5*(q.v[3].y + q.v[2].y); v32.z = 0.5*(q.v[3].z +q.v[2].z); sub0.v[0] = q.v[0]; sub0.v[1] = v01; sub0.v[2] = v32;sub0.v[3] = q.v[3]; sub1.v[0] = v01; sub1.v[1] = q.v[1]; sub1.v[2] =q.v[2]; sub1.v[3] = v32; } void Tess: :divideArc (const Quad& q,Quad&sub0 ,Quad& sub1) { sub0.v[0] = q.v[0]; sub0.v[1] = v01; sub0.v[2] =v32; sub0.v[3] = q.v[3]; sub1.v[0] = v01; sub1.v[1] = q.v[1]; sub1.v[2]= q.v[2]; sub1.v[3] = v32; } void Tess: :divideArea (const Quad& q,Quad&sub0 ,Quad& sub1) { double w = 0.5; double delta = 0.25; do { constVector3 v01 = slerp(q.v[0],q.v[1],w); const Vector3 v32 =slerp(q.v[3],q.v[2],w); sub0.v[0] = q.v[0]; sub0.v[1] = v01; sub0.v[2] =v32; sub0.v[3] = q.v[3]; sub1.v[0] = v01; sub1.v[1] = q.v[1]; sub1.v[2]= q.v[2]; sub1.v[3] = v32; const double area0 = sub0.sarea(); constdouble area1 = sub1.sarea(); if ( area0 < area1 ) w += delta; else if (area0 > area1 ) w −= delta; else // exact match break; // update thesearch variable delta *= 0.5; } while ( delta > std::numeric_limits<double>: :epsilon() ) ;} FILENAME: vector3.cpp #include<cmath> #include ″vector3.h″ // constructor Vector3: :Vector3(constdouble _x,const double _y,const double _z) : x(_x), y(_y), z(_z) { } //dot product double Vector3: :dot(const Vector3& v) const { returnx*v.x + y*v.y + z*v.z; } // normalize void Vector3: :normalize (void) {// get the length of the vector const double length = sqrtf(dot(*this));// if the length is non-zero, divide by the length if ( length != 0.0f ){ // use the reciprocal for efficiency const double rlength =1.0f/length; x *= rlength; y *= rlength; z *= rlength; } } // write tostream std: :ostream& operator<<(std: :ostream& ostr,const Vector3&v) {ostr << ″(″ << v.x << ″,″ << v.y << ″,″ << v.z << ″)″; return ostr; } //cross product Vector3 operator*(const Vector3& a,const Vector3& b) {return Vector3 a.y*b.z − a.z*b.y, a.z*b.x − a.x*b.z, a.x*b.y − a.y*b.x);} // spherical interpolation Vector3 slerp(const Vector3& a,constVector3& b,double weight) { // special case // get normalized inputvectors Vector3 an = a; Vector3 bn = b; an.normalize(); bn.normalize();// determine the angle between the vectors const double dotab =an.dot(bn); const double theta = acosf (dotab); // check forantiparallel vectors if ( dotab == −1.0 ) throw 0; // compute thecartesian interpolation factor that // corresponds to this angularinterpolation factor const double cweight = (theta != 0.0) ? 0.5*(1.0 +tan(theta*(weight − 0.5))/tan(0.5*theta)) : 0.0; // do cartesianinterpolation Vector3 c( an.x*(1.0 − cweight) + bn.x*cweight, an.y*(1.0− cweight) + bn.y*cweight, an.z*(1.0 − cweight) + bn.z*cweight); //normalize the result c.normalize(); return c; } FILENAME: vector3.h#ifndef _VECTOR3_H_ #def ine _VECTOR3_H_ #include <iostream> //: basic3D vector struct Vector3 { double x,y,z; // vector components //:default constructor Vector3(void) { } //: constructor with initialvalues Vector3(const double xx,const double yy,const double zz): //: dotproduct double dot(const Vector3& v) const; //: normalize // This setsthe vector to unit length void normalize (void); }; //: i/o std::ostream& operator<<(std: :ostream& ostr,const Vector3& v); //: crossproduct Vector3 operator*(const Vector3& a,const Vector3& b); //:spherical interpolation // This interpolates directions angularly toyield a unit // vector. A weight of 0 returns a (normalized); a weight// of 1 returns b (normalized). Values of weight in between // 0 and 1return a unit vector interpolated linearly with // respect to angle.Values outside [0,1] are undefined. Vector3 slerp(const Vector3& a,constVector3& b,double weight = 0.5); #endif // _VECTOR3_H_(—) FILENAME:quad.h #ifndef _QUAD_H_ #define _QUAD_H_ #include ″vector3.h″ //:quadrilateral struct Quad { Vector3 v[4]; // vertices //: defaultconstructor Quad(void) { }; //: constructor Quad( const Vector3& v0,const Vector3& v1, const Vector3& v2, const Vector3& v3); //: solidangle // This gives the solid angle subtended by the // quad as viewedfrom the origin. No vertex can // be located at the origin. doublesarea(void) const; //: arc length ratio // This returns the ratio of thelongest arc to the // shortest arc subtended by the quad as viewed from// the origin. No vertex can be located at the origin. doublearatio(void) const; //: rotate vertices // This moves v1 to v0, v2 tov1, etc. It is useful // for subdivision along alternating axes. voidrotateVertices(void); }; //: write to stream std: :ostream&operator<<(std: :ostream& ostr,const Quad& q); #endif // _QUAD_H_(—)FILENAME:quad.cpp #include <cmath> #ifndef M_PI #define M_PI3.1415926535897932385 #endif // M_PI #include ″quad.h″ // constructorQuad: :Quad( const Vector3& v0, const Vector3& v1, const Vector3& v2,const Vector3& v3) { v[0] = v0; v[1] = v1; v[2] = v2; v[3] = v3; } //solid angle double Quad: :sarea(void) const { // This macro creates aunit yector called NAME that is the tangent // on the sphere of the arcbetween VERTEXA and VERTEXB, at // VERTEXA. The tangent will point alongthe arc toward VERTEXB. #define TANGENT (NAME, VERTEXA, VERTEXB) \Vector3 NAME = v[VERTEXA]*v[VERTEXB] *v[VERTEXA]; \ NAME.normalize ()TANGENT(t0a,0,3); TANGENT(t1a,1,0); TANGENT(t2a,2,1); TANGENT(t3a,3,2);TANGENT(t0b,0,1); TANGENT(t1b,1,2); TANGENT(t2b,2,3); TANGENT(t3b,3,0);// vertex angles are the inverse cosine of the dot product const doublea0 = acos(t0a.dot(t0b)); const double a1 = acos(t1a.dot(t1b)); constdouble a2 = acos(t2a.dot(t2b)); const double a3 = acos(t3a.dot(t3b)); //solid angle is the sum of the vertex angles − 2PI const double solid =a0 + a1 + a2 + a3 − 2.0*M_PI; return solid; } // arc ratio // This isthe ratio of the largest arc length to the smallest arc length doubleQuad: :aratio(void) const { // create unit vectors Vector3 t0 = v[0];t0.normalize(); Vector3 t1 = v[1]; t1.normalize(); Vector3 t2 = v[2];t2.normalize(); Vector3 t3 = v[3]; t3.normalize(); // determine arclengths double minLength = M_PI; double maxLength = 0.0; #defineARC(I,J) { / const double length = acos(t##I.dot(t##J)); \ if ( length <minLength ) minLength = length; \ if ( length > maxLength ) maxLength =length; \ } ARC(0,1); ARC(1,2); ARC(2,3); ARC(3,0); returnmaxLength/minLength; } // rotate vertices against the indices void Quad::rotateVertices (void) { Vector3 tmp = v[0]; v[0]= v[1]; v[1]= v[2];v[2]= v[3]; v[3]= tmp; } // write to stream std: :ostream&operator<<(std: :ostream& ostr,const Quad& q) { ostr << ″[″ << q.v[0] <<″,″ << q.v[1] << ″,″ << q.v[2] << ″,″ << q.v[3] << ″]″; return ostr; }

What is claimed is:
 1. A method for creating an environment map from oneor more images representing an environment, the method comprising:creating a texture map having a plurality of last-generation polygonalcurved surfaces as by selecting a plurality of initial polygonal curvedsurfaces; and dividing each initial polygonal curved surface to form aplurality of second-generation polygonal curved surfaces; determining animage area in the one or more images corresponding to.each polygonalcurved surface; and coloring each polygonal curved surface based on thecorresponding area.
 2. The method of claim 1, further comprisingdividing each second-generation polygonal curved surface to form aplurality of third-generation polygonal curved surfaces.
 3. The methodof claim 1, wherein each second-generation polygonal curved surface hasan equal area.
 4. The method of claim 1, wherein each of the secondgeneration polygonal curved surfaces has N sides, and wherein each ofthe second-generation polygonal curved surfaces has a common side with Nother second-generation polygonal curved surfaces.
 5. The method ofclaim 1, wherein a first vertex of each second-generation polygonalcurved surface is at a midpoint of a first side of an initial polygonalcurved surface, and wherein a second vertex of the second-generationpolygonal curved surface is at a midpoint of a second side of theinitial polygonal curved surface.
 6. The method of claim 1, furthercomprising converting the last-generation polygonal curved surface ofeach initial polygonal curved surface into a two-dimensional polygonalimage to form a-plurality of two-dimensional polygonal images; andconcatenating the plurality of two-dimensional polygonal images to formthe environment map.
 7. The method of claim 1, wherein each facet has aresolution of a single pixel.
 8. The method of claim 1, whereindetermining an image area in the one or more images corresponding toeach polygonal curved surface further comprises: determining a solidangle projection for each facet; and calculating the image area as theintersection of the solid angle with the one or more images.
 9. Themethod of claim 1, wherein said polygonal curved surfaces have aspherical base curve.
 10. The method of claim 1, wherein eachlast-generation polygonal curved surface is a tetragonal curved surface.11. An environment map creation system for creating an environment mapfrom one or more images representing an environment, the environment mapcreation system comprising: an environment map rendering unit configuredto receive the one or more images; and a texture projection generationunit coupled to the environment map rendering unit and configured toproduce a texture projection comprising a plurality of initial polygonalcurved surfaces divided into a plurality of last-generation polygonalcurved surfaces, wherein each last generation polygonal curved surfaceis a tetragonal curved surface; wherein the environment map renderingunit creates the environment map using the texture projection.
 12. Theenvironment map creation system of claim 11, wherein eachlast-generation polygonal curved surface has an equal area.
 13. Theenvironment map creation system of claim 11, wherein in the environmentmap rendering unit is configured to convert the plurality oflast-generation polygonal curved surfaces of each initial polygonalcurved surface into a two-dimensional polygonal image to form aplurality of two-dimensional polygonal images.
 14. The environmentcreation system of claim 13, wherein the map rendering unit isconfigured to concatenate the plurality of two-dimensional polygonalimages to form the environment map.